Doing code reviews is a great way to discover things that people might
struggle to comprehend. While proof-reading
OpenStack patches recently, I spotted that
people were not using correctly the various decorators Python provides for
methods. So here's my attempt at providing me a link to send them to in my
next code reviews. :-)
How methods work in Python
A method is a function that is stored as a class attribute. You can declare
and access such a function this way:
>>> class Pizza(object):
... def __init__(self, size):
... self.size = size
... def get_size(self):
... return self.size
...
>>> Pizza.get_size
<unbound method Pizza.get_size>
What Python tells you here, is that the attribute
get_size of the class
Pizza is a method that is unbound. What does this mean? We'll know as
soon as we'll try to call it:
>>> Pizza.get_size()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method get_size() must be called with Pizza instance as first argument (got nothing instead)
We can't call it because it's not bound to any instance of
Pizza. And a
method wants an instance as its first argument (in Python 2 it must be
an instance of that class; in Python 3 it could be anything). Let's try to
do that then:
>>> Pizza.get_size(Pizza(42))
42
It worked! We called the method with an instance as its first argument, so
everything's fine. But you will agree with me if I say this is not a very
handy way to call methods; we have to refer to the class each time we want
to call a method. And if we don't know what class is our object, this is not
going to work for very long.
So what Python does for us, is that it binds all the methods from the class
Pizza to any instance of this class. This means that the attribute
get_size of an instance of
Pizza is a bound method: a method for
which the first argument will be the instance itself.
>>> Pizza(42).get_size
<bound method Pizza.get_size of <__main__.Pizza object at 0x7f3138827910>>
>>> Pizza(42).get_size()
42
As expected, we don't have to provide any argument to
get_size, since
it's bound, its
self argument is automatically set to our
Pizza
instance. Here's a even better proof of that:
>>> m = Pizza(42).get_size
>>> m()
42
Indeed, you don't even have to keep a reference to your
Pizza object. Its
method is bound to the object, so the method is sufficient to itself.
But what if you wanted to know which object this bound method is bound to?
Here's a little trick:
>>> m = Pizza(42).get_size
>>> m.__self__
<__main__.Pizza object at 0x7f3138827910>
>>> # You could guess, look at this:
...
>>> m == m.__self__.get_size
True
Obviously, we still have a reference to our object, and we can find it back
if we want.
In Python 3, the functions attached to a class are not considered as
unbound method anymore, but as simple functions, that are bound to an
object if required. So the principle stays the same, the model is just
simplified.
>>> class Pizza(object):
... def __init__(self, size):
... self.size = size
... def get_size(self):
... return self.size
...
>>> Pizza.get_size
<function Pizza.get_size at 0x7f307f984dd0>
Static methods
Static methods are a special case of methods. Sometimes, you'll write code
that belongs to a class, but that doesn't use the object itself at all. For
example:
class Pizza(object):
@staticmethod
def mix_ingredients(x, y):
return x + y
def cook(self):
return self.mix_ingredients(self.cheese, self.vegetables)
In such a case, writing
mix_ingredients as a non-static method would work
too, but it would provide it a
self argument that would not be used. Here,
the decorator
@staticmethod buys us several things:
- Python doesn't have to instantiate a bound-method for each Pizza object
we instiantiate. Bound methods are objects too, and creating them has a
cost. Having a static method avoids that:
>>> Pizza().cook is Pizza().cook
False
>>> Pizza().mix_ingredients is Pizza.mix_ingredients
True
>>> Pizza().mix_ingredients is Pizza().mix_ingredients
True
-
It eases the readability of the code: seeing @staticmethod, we know that
the method does not depend on the state of object itself;
-
It allows us to override the mix_ingredients method in a subclass. If we
used a function mix_ingredients defined at the top-level of our module, a
class inheriting from Pizza wouln't be able to change the way we mix
ingredients for our pizza without overriding cook itself.
Class methods
Having said that, what are class methods? Class methods are methods that are
not bound to an object, but to a class!
>>> class Pizza(object):
... radius = 42
... @classmethod
... def get_radius(cls):
... return cls.radius
...
>>>
>>> Pizza.get_radius
<bound method type.get_radius of <class '__main__.Pizza'>>
>>> Pizza().get_radius
<bound method type.get_radius of <class '__main__.Pizza'>>
>>> Pizza.get_radius is Pizza().get_radius
True
>>> Pizza.get_radius()
42
Whatever the way you use to access this method, it will be always bound to
the class it is attached too, and its first argument will be the class
itself (remember that classes are objects too).
When to use this kind of methods? Well class methods are mostly useful for
two types of methods:
- Factory methods, that are used to create an instance for a class using for
example some sort of pre-processing. If we use a @staticmethod instead, we
would have to hardcode the Pizza class name in our function, making any
class inheriting from Pizza unable to use our factory for its own use.
class Pizza(object):
def __init__(self, ingredients):
self.ingredients = ingredients
@classmethod
def from_fridge(cls, fridge):
return cls(fridge.get_cheese() + fridge.get_vegetables())
- Static methods calling static methods: if you split a static methods in
several static methods, you shouldn't hard-code the class name but use class
methods. Using this way to declare ou method, the Pizza name is never
directly
referenced and inheritance and method overriding will work flawlessly
class Pizza(object):
def __init__(self, radius, height):
self.radius = radius
self.height = height
@staticmethod
def compute_circumference(radius):
return math.pi * (radius ** 2)
@classmethod
def compute_volume(cls, height, radius):
return height * cls.compute_circumference(radius)
def get_volume(self):
return self.compute_volume(self.height, self.radius)
Abstract methods
An abstract method is a method defined in a base class, but that may not
provide any implementation. In Java, it would describe the methods of an
interface.
So the simplest way to write an abstract method in Python is:
class Pizza(object):
def get_radius(self):
raise NotImplementedError
Any class inheriting from
Pizza should implement and override the
get_radius method, otherwise an exception would be raised.
This particular way of implementing abstract method has a drawback. If you
write a class that inherits from
Pizza and forget to implement
get_radius, the error will only be raised when you'll try to use that
method.
>>> Pizza()
<__main__.Pizza object at 0x7fb747353d90>
>>> Pizza().get_radius()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in get_radius
NotImplementedError
There's a way to triggers this way earlier, when the object is being
instantiated, using the
abc
module that's provided with Python.
import abc
class BasePizza(object):
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def get_radius(self):
"""Method that should do something."""
Using
abc and its special class, as soon as you'll try to instantiate
BasePizza or any class inheriting from it, you'll get a
TypeError.
>>> BasePizza()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class BasePizza with abstract methods get_radius
Mixing static, class and abstract methods
When building classes and inheritances, the time will come where you will
have to mix all these methods decorators. So here's some tips about it.
Keep in mind that declaring a class as being abstract, doesn't freeze the
prototype of that method. That means that it must be implemented, but i can
be implemented with any argument list.
import abc
class BasePizza(object):
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def get_ingredients(self):
"""Returns the ingredient list."""
class Calzone(BasePizza):
def get_ingredients(self, with_egg=False):
egg = Egg() if with_egg else None
return self.ingredients + egg
This is valid, since
Calzone fulfil the interface requirement we defined
for
BasePizza objects. That means that we could also implement it as being
a class or a static method, for example:
import abc
class BasePizza(object):
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def get_ingredients(self):
"""Returns the ingredient list."""
class DietPizza(BasePizza):
@staticmethod
def get_ingredients():
return None
This is also correct and fulfil the contract we have with our abstract
BasePizza class. The fact that the
get_ingredients method don't need to know
about the object to return result is an implementation detail, not a
criteria to have our contract fulfilled.
Therefore, you can't force an implementation of your abstract method to be a
regular, class or static method, and arguably you shouldn't. Starting with
Python 3 (this won't work as you would expect in Python 2, see
issue5867), it's now possible to use the
@staticmethod and
@classmethod decorators on top of
@abstractmethod:
import abc
class BasePizza(object):
__metaclass__ = abc.ABCMeta
ingredient = ['cheese']
@classmethod
@abc.abstractmethod
def get_ingredients(cls):
"""Returns the ingredient list."""
return cls.ingredients
Don't misread this: if you think this going to force your subclasses to
implement
get_ingredients as a class method, you are wrong. This simply
implies that your implementation of
get_ingredients in the
BasePizza
class is a class method.
An implementation in an abstract method? Yes! In Python, contrary to methods
in Java interfaces, you can have code in your abstract methods and call it
via
super():
import abc
class BasePizza(object):
__metaclass__ = abc.ABCMeta
default_ingredients = ['cheese']
@classmethod
@abc.abstractmethod
def get_ingredients(cls):
"""Returns the ingredient list."""
return cls.default_ingredients
class DietPizza(BasePizza):
def get_ingredients(self):
return ['egg'] + super(DietPizza, self).get_ingredients()
In such a case, every pizza you will build by inheriting from
BasePizza
will have to override the
get_ingredients method, but will be able to use
the default mechanism to get the ingredient list by using
super().
The week has been wonderful. It was my second OpenStack design summit, and I
loved it. Bumping into various people I've never met so far and worked with
online was a real pleasure. As it was to meet again with fellow OpenStack
developers! The event organisation was great, as were the parties. :-)
On the last day, I had the chance to present a talk with Eoghan Glynn and
Nick Barcet how we built the auto-scaling feature in Heat, implementing the
"alarming" feature needed in Ceilometer.
A huge release, the combined 0.9.9 and 0.9.10 releases (I made a mistake releasing) are now tagged and pushed to pypi. It features a number of enhancements and fixes, and is just an absolute thrill to play with.
Thanks to the contributors this cycle:
I always looked at Github from a distant eye, mainly because I always
disliked their pull-request handling, and saw no value in the social hype it
brings. Why?
One click away isn't one click effort
The pull-request system looks like an incredible easy way to contribute to
any project hosted on Github. You're a click away to send your contribution
to any software. But the problem is that any worthy contribution isn't an
effort of a single click.
Doing any proper and useful contribution to a software is never done right
the first time. There's a dance you will have to play. A slowly rhythmed
back and forth between you and the software maintainer or team. You'll have
to dance it until your contribution is correct and can be merged.
But as a software maintainer, not everybody is going to follow you on this
choregraphy, and you'll end up with pull-request you'll never get finished
unless you wrap things up yourself. So the gain in pull-requests here, isn't
really bigger than a good bug report in most cases.
This is where the social argument of Github isn't anymore. As soon as you're
talking about projects bigger than a color theme for your favorite text
editor, this feature is overrated.
Contribution rework
If you're lucky enough, your contributor will play along and follow you on
this pull-request review process. You'll make suggestions, he will listen
and will modify his pull-request to follow your advice.
At this point, there's two technics he can use to please you.
Technic #1: the Topping
Github's pull-requests invite you to send an entire branch, eclipsing the
fact that it is composed of several commits. The problem is that a lot of
one-click-away contributors do not masterize Git and/or do not make efforts
to build a logical patchset, and nothing warns them that their branch
history is wrong. So they tend to change stuff around, commit, make a
mistake, commit, fix this mistake, commit, etc. This kind of branch is
composed of the whole brain's construction process of your contributor, and
is a real pain to review. To the point I quite often give up.
<figure>
<figcaption>
A typical case: 3 commits to build a 4 lines long file.
</figcaption>
</figure>
Without Github, the old method that all software used, and that many
software still use (e.g. Linux), is to send a patch set over e-mail (or any
other medium like Gerrit). This method has one positive effect, that it
forces the contributor to acknowledge the list of commits he is going to
publish. So, if the contributor he has fixup commits in his history, they
are going to be seen as first class citizen. And nobody is going to want to
see that, neither your contributor, nor the software maintainers. Therefore,
such a system tend to push contributors to write atomic, logical and
self-contained patchset that can be more easily reviewed.
Technic #2: the History Rewriter
This is actually the good way to build a working and logical patchset using
Git. Rewriting history and amending problematic patches using the famous
git rebase --interactive trick.
The problem is that if your contributor does this and then repush the branch
composing your pull-request to Github, you will both lose the previous
review done, each time. There's no history on the different versions of the
branch that has been pushed.
In the old alternative system like e-mail, no information is lost when
reworked patches are resent, obviously. This is far better because it eases
the following of the iterative discussions that the patch triggered.
Of course, it would be possible for Github to enhance this and fix it, but
currently it doesn't handle this use case correctly..
<figure>
It's not a secret for anyone that I've been contributing to OpenStack as a
daily routine for the last 18 months. The more I contribute, the more I like
the contribution workflow and process. It's already
The summit has been wonderful. It was my first OpenStack design summit --
even more as a PTL -- and bumping into various people I've never met so far
and worked with online only was a real pleasure!
<figure>
<figcaption>Me and Nick ready to talk about Ceilometer new features.</figcaption>
</figure>
<figcaption>Doug explaining Ceilometer architecture.</figcaption>
</figure>
The next session was about getting feedbacks from our users. We had a lot of
surprise to discover wonderful real use-cases and deployments, like the CERN
using Ceilometer and generating 2 GB of data per day!
The following sessions ran on Thursday and were much more about new features
discussion. A lot ot already existing blueprints were discussed and quickly
validated during the first morning session. Then,
I actually started to hack a bit on Hy, and will try to continue to do so,
since I'm really eager to learn a bit more about both Lisp and Python
internals in the process. I've already send a few patches on small bugs I've
encountered, and proposed a few ideas. It's really exciting to be able to
influence early a language design that I'll love to use! Being a recent fan
of Common Lisp, I tend to grab the good stuff from it to add them into Hy.
